Distributing and accessing data in a data processing system

ABSTRACT

A method of distributing a data item to a plurality of computers in a network of computers. A data item is divided into a plurality of segments. Each segment is stored on at least one computer in said network. A list of computers onto which each of said segments has been stored is maintained. When a data item is requested, its component segments are obtained from at least two computers in the network. Some of the computers may form a peer-to-peer network.

RELATED APPLICATIONS

This application is a continuation of an claims priority to pending U.S.patent application Ser. No. 11/724,232, which is a continuation ofco-pending application Ser. No. 11/017,650, filed Dec. 22, 2004, whichis a continuation of pending application Ser. No. 10/742,972, filed Dec.23, 2003, which is a continuation of Ser. No. 09/987,723, filed Nov. 15,2001, patented as U.S. Pat. No. 6,928,442; which is a which is acontinuation of application Ser. No. 09/283,160, filed Apr. 1, 1999, nowU.S. Pat. No. 6,415,280, which is a division of application Ser. No.08/960,079, filed Oct. 24, 1997, now U.S. Pat. No. 5,978,791, which is acontinuation of Ser. No. 08/425,160, filed Apr. 11, 1995, now abandoned,the contents of which each of these applications are hereby incorporatedherein by reference. This application is a continuation of and claimspriority to co-pending application Ser. No. 11/017,650, filed Dec. 22,2004, which is a continuation of application Ser. No. 09/987,723, filedNov. 15, 2001, now U.S. Pat. No. 6,928,442, which is a continuation ofapplication Ser. No. 09/283,160, filed Apr. 1, 1999, now U.S. Pat. No.6,415,280, which is a division of application Ser. No. 08/960,079, filedOct. 24, 1997, now U.S. Pat. No. 5,978,791, which is a continuation ofSer. No. 08/425,160, filed Apr. 11, 1995, now abandoned, the contents ofwhich each of these applications are hereby incorporated herein byreference. This is also a continuation of and claims priority toco-pending application Ser. No. 10/742,972, filed Dec. 23, 2003, whichis a division of application Ser. No. 09/987,723, filed Nov. 15, 2001,now U.S. Pat. No. 6,928,442, which is a continuation of application Ser.No. 09/283,160, filed Apr. 1, 1999, now U.S. Pat. No. 6,415,280, whichis a division of application Ser. No. 08/960,079, filed Oct. 24, 1997,now U.S. Pat. No. 5,978,791, which is a continuation of Ser. No.08/425,160, filed Apr. 11, 1995, now abandoned, the contents of whicheach of these applications are hereby incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to data processing systems and, moreparticularly, to data processing systems wherein data items areidentified by substantially unique identifiers which depend on all ofthe data in the data items and only on the data in the data items.

2. Background of the Invention

Data processing (DP) systems, computers, networks of computers, or thelike, typically offer users and programs various ways to identify thedata in the systems.

Users typically identify data in the data processing system by givingthe data some form of name. For example, a typical operating system (OS)on a computer provides a file system in which data items are named byalphanumeric identifiers. Programs typically identify data in the dataprocessing system using a location or address. For example, a programmay identify a record in a file or database by using a record numberwhich serves to locate that record.

In all but the most primitive operating systems, users and programs areable to create and use collections of named data items, thesecollections themselves being named by identifiers. These namedcollections can then, themselves, be made part of other namedcollections. For example, an OS may provide mechanisms to group files(data items) into directories (collections). These directories can then,themselves be made part of other directories. A data item may thus beidentified relative to these nested directories using a sequence ofnames, or a so-called pathname, which defines a path through thedirectories to a particular data item (file or directory).

As another example, a database management system may group data records(data items) into tables and then group these tables into database files(collections). The complete address of any data record can then bespecified using the database file name, the table name, and the recordnumber of that data record.

Other examples of identifying data items include: identifying files in anetwork file system, identifying objects in an object-oriented database,identifying images in an image database, and identifying articles in atext database.

In general, the terms “data” and “data item” as used herein refer tosequences of bits. Thus a data item may be the contents of a file, aportion of a file, a page in memory, an object in an object-orientedprogram, a digital message, a digital scanned image, a part of a videoor audio signal, or any other entity which can be represented by asequence of bits. The term “data processing” herein refers to theprocessing of data items, and is sometimes dependent on the type of dataitem being processed. For example, a data processor for a digital imagemay differ from a data processor for an audio signal.

In all of the prior data processing systems the names or identifiersprovided to identify data items (the data items being files,directories, records in the database, objects in object-orientedprogramming, locations in memory or on a physical device, or the like)are always defined relative to a specific context. For instance, thefile identified by a particular file name can only be determined whenthe directory containing the file (the context) is known. The fileidentified by a pathname can be determined only when the file system(context) is known. Similarly, the addresses in a process address space,the keys in a database table, or domain names on a global computernetwork such as the Internet are meaningful only because they arespecified relative to a context.

In prior art systems for identifying data items there is no directrelationship between the data names and the data item. The same dataname in two different contexts may refer to different data items, andtwo different data names in the same context may refer to the same dataitem.

In addition, because there is no correlation between a data name and thedata it refers to, there is no a priori way to confirm that a given dataitem is in fact the one named by a data name. For instance, in a DPsystem, if one processor requests that another processor deliver a dataitem with a given data name, the requesting processor cannot, ingeneral, verify that the data delivered is the correct data (given onlythe name). Therefore it may require further processing, typically on thepart of the requester, to verify that the data item it has obtained is,in fact, the item it requested.

A common operation in a DP system is adding a new data item to thesystem. When a new data item is added to the system, a name can beassigned to it only by updating the context in which names are defined.Thus such systems require a centralized mechanism for the management ofnames. Such a mechanism is required even in a multi-processing systemwhen data items are created and identified at separate processors indistinct locations, and in which there is no other need forcommunication when data items are added.

In many data processing systems or environments, data items aretransferred between different locations in the system. These locationsmay be processors in the data processing system, storage devices,memory, or the like. For example, one processor may obtain a data itemfrom another processor or from an external storage device, such as afloppy disk, and may incorporate that data item into its system (usingthe name provided with that data item).

However, when a processor (or some location) obtains a data item fromanother location in the DP system, it is possible that this obtaineddata item is already present in the system (either at the location ofthe processor or at some other location accessible by the processor) andtherefore a duplicate of the data item is created. This situation iscommon in a network data processing environment where proprietarysoftware products are installed from floppy disks onto severalprocessors sharing a common file server. In these systems, it is oftenthe case that the same product will be installed on several systems, sothat several copies of each file will reside on the common file server.

In some data processing systems in which several processors areconnected in a network, one system is designated as a cache server tomaintain master copies of data items, and other systems are designatedas cache clients to copy local copies of the master data items into alocal cache on an as-needed basis. Before using a cached item, a cacheclient must either reload the cached item, be informed of changes to thecached item, or confirm that the master item corresponding to the cacheditem has not changed. In other words, a cache client must synchronizeits data items with those on the cache server. This synchronization mayinvolve reloading data items onto the cache client. The need to keep thecache synchronized or reload it adds significant overhead to existingcaching mechanisms.

In view of the above and other problems with prior art systems, it istherefore desirable to have a mechanism which allows each processor in amultiprocessor system to determine a common and substantially uniqueidentifier for a data item, using only the data in the data item and notrelying on any sort of context.

It is further desirable to have a mechanism for reducing multiple copiesof data items in a data processing system and to have a mechanism whichenables the identification of identical data items so as to reducemultiple copies. It is further desirable to determine whether twoinstances of a data item are in fact the same data item, and to performvarious other systems' functions and applications on data items withoutrelying on any context information or properties of the data item.

It is also desirable to provide such a mechanism in such a way as tomake it transparent to users of the data processing system, and it isdesirable that a single mechanism be used to address each of theproblems described above.

SUMMARY OF THE INVENTION

This invention provides, in a data processing system, a method andapparatus for identifying a data item in the system, where the identityof the data item depends on all of the data in the data item and only onthe data in the data item. Thus the identity of a data item isindependent of its name, origin, location, address, or other informationnot derivable directly from the data, and depends only on the dataitself.

This invention further provides an apparatus and a method fordetermining whether a particular data item is present in the system orat a location in the system, by examining only the data identities of aplurality of data items.

Using the method or apparatus of the present invention, the efficiencyand integrity of a data processing system can be improved. The presentinvention improves the design and operation of a data storage system,file system, relational database, object-oriented database, or the likethat stores a plurality of data items, by making possible or improvingthe design and operation of at least some or all of the followingfeatures:

the system stores at most one copy of any data item at a given location,even when multiple data names in the system refer to the same contents;

the system avoids copying data from source to destination locations whenthe destination locations already have the data;

the system provides transparent access to any data item by referenceonly to its identity and independent of its present location, whether itbe local, remote, or offline;

the system caches data items from a server, so that only the mostrecently accessed data items need be retained;

when the system is being used to cache data items, problems ofmaintaining cache consistency are avoided;

the system maintains a desired level of redundancy of data items in anetwork of servers, to protect against failure by ensuring that multiplecopies of the data items are present at different locations in thesystem;

the system automatically archives data items as they are created ormodified;

the system provides the size, age, and location of groups of data itemsin order to decide whether they can be safely removed from a local filesystem;

the system can efficiently record and preserve any collection of dataitems;

the system can efficiently make a copy of any collection of data items,to support a version control mechanism for groups of the data items;

the system can publish data items, allowing other, possibly anonymous,systems in a network to gain access to the data items and to rely on theavailability of the data items;

the system can maintain a local inventory of all the data items locatedon a given removable medium, such as a diskette or CD-ROM, the inventoryis independent of other properties of the data items such as their name,location, and date of creation;

the system allows closely related sets of data items, such as matchingor corresponding directories on disconnected computers, to beperiodically resynchronized with one another;

the system can verify that data retrieved from another location is thedesired or requested data, using only the data identifier used toretrieve the data;

the system can prove possession of specific data items by contentwithout disclosing the content of the data items, for purposes of laterlegal verification and to provide anonymity;

the system tracks possession of specific data items according to contentby owner, independent of the name, date, or other properties of the dataitem, and tracks the uses of specific data items and files by contentfor accounting purposes.

Other objects, features, and characteristics of the present invention aswell as the methods of operation and functions of the related elementsof structure, and the combination of parts and economies of manufacture,will become more apparent upon consideration of the followingdescription and the appended claims with reference to the accompanyingdrawings, all of which form a part of this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1(a) and 1(b) depict a typical data processing system in which apreferred embodiment of the present invention operates;

FIG. 2 depicts a hierarchy of data items stored at any location in sucha data processing system;

FIGS. 3-9 depict data structures used to implement an embodiment of thepresent invention; and

FIGS. 10(a)-28 are flow charts depicting operation of various aspects ofthe present invention.

DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EXEMPLARY EMBODIMENTS

An embodiment of the present invention is now described with referenceto a typical data processing system 100, which, with reference to FIGS.1(a) and 1(b), includes one or more processors (or computers) 102 andvarious storage devices 104 connected in some way, for example by a bus106.

Each processor 102 includes a CPU 108, a memory 110 and one or morelocal storage devices 112. The CPU 108, memory 110, and local storagedevice 112 may be internally connected, for example by a bus 114. Eachprocessor 102 may also include other devices (not shown), such as akeyboard, a display, a printer, and the like.

In a data processing system 100, wherein more than one processor 102 isused, that is, in a multiprocessor system, the processors may be in oneof various relationships. For example, two processors 102 may be in aclient/server, client/client, or a server/server relationship. Theseinter-processor relationships may be dynamic, changing depending onparticular situations and functions. Thus, a particular processor 102may change its relationship to other processors as needed, essentiallysetting up a peer-to-peer relationship with other processors. In apeer-to-peer relationship, sometimes a particular processor 102 acts asa client processor, whereas at other times the same processor acts as aserver processor. In other words, there is no hierarchy imposed on orrequired of processors 102.

In a multiprocessor system, the processors 102 may be homogeneous orheterogeneous. Further, in a multiprocessor data processing system 100,some or all of the processors 102 may be disconnected from the networkof processors for periods of time. Such disconnection may be part of thenormal operation of the system 100 or it may be because a particularprocessor 102 is in need of repair.

Within a data processing system 100, the data may be organized to form ahierarchy of data storage elements, wherein lower level data storageelements are combined to form higher level elements. This hierarchy canconsist of, for example, processors, file systems, regions, directories,data files, segments, and the like. For example, with reference to FIG.2, the data items on a particular processor 102 may be organized orstructured as a file system 116 which comprises regions 117, each ofwhich comprises directories 118, each of which can contain otherdirectories 118 or files 120. Each file 120 being made up of one or moredata segments 122.

In a typical data processing system, some or all of these elements canbe named by users given certain implementation specific namingconventions, the name (or pathname) of an element being relative to acontext. In the context of a data processing system 100, a pathname isfully specified by a processor name, a filesystem name, a sequence ofzero or more directory names identifying nested directories, and a finalfile name. (Usually the lowest level elements, in this case segments122, cannot be named by users.)

In other words, a file system 116 is a collection of directories 118. Adirectory 118 is a collection of named files 120—both data files 120 andother directory files 118. A file 120 is a named data item which iseither a data file (which may be simple or compound) or a directory file118. A simple file 120 consists of a single data segment 122. A compoundfile 120 consists of a sequence of data segments 122. A data segment 122is a fixed sequence of bytes. An important property of any data segmentis its size, the number of bytes in the sequence.

A single processor 102 may access one or more file systems 116, and asingle storage device 104 may contain one or more file systems 116, orportions of a file system 116. For instance, a file system 116 may spanseveral storage devices 104.

In order to implement controls in a file system, file system 116 may bedivided into distinct regions, where each region is a unit of managementand control. A region consists of a given directory 118 and isidentified by the pathname (user defined) of the directory.

In the following, the term “location”, with respect to a data processingsystem 100, refers to any of a particular processor 102 in the system, amemory of a particular processor, a storage device, a removable storagemedium (such as a floppy disk or compact disk), or any other physicallocation in the system. The term “local” with respect to a particularprocessor 102 refers to the memory and storage devices of thatparticular processor.

In the following, the terms “True Name”, “data identity” and “dataidentifier” refer to the substantially unique data identifier for aparticular data item. The term “True File” refers to the actual file,segment, or data item identified by a True Name.

A file system for a data processing system 100 is now described which isintended to work with an existing operating system by augmenting some ofthe operating system's file management system codes. The embodimentprovided relies on the standard file management primitives for actuallystoring to and retrieving data items from disk, but uses the mechanismsof the present invention to reference and access those data items.

The processes and mechanisms (services) provided in this embodiment aregrouped into the following categories: primitive mechanisms, operatingsystem mechanisms, remote mechanisms, background mechanisms, andextended mechanisms.

Primitive mechanisms provide fundamental capabilities used to supportother mechanisms. The following primitive mechanisms are described:

1. Calculate True Name;

2. Assimilate Data Item;

3. True File;

4. Get True Name from Path;

5. Link path to True Name;

6. Realize True File from Location;

7. Locate Remote File;

8. Make True File Local;

9. Create Scratch File;

10. Freeze Directory;

11. Expand Frozen Directory;

12. Delete True File;

13. Process Audit File Entry;

14. Begin Grooming;

15. Select For Removal; and

16. End Grooming.

Operating system mechanisms provide typical familiar file systemmechanisms, while maintaining the data structures required to offer themechanisms of the, present invention. Operating system mechanisms aredesigned to augment existing operating systems, and in this way to makethe present invention compatible with, and generally transparent to,existing applications. The following operating system mechanisms aredescribed:

1. Open File;

2. Close File;

3. Read File;

4. Write File;

5. Delete File or Directory;

6. Copy File or Directory;

7. Move File or Directory;

8. Get File Status; and

9. Get Files in Directory.

Remote mechanisms are used by the operating system in responding torequests from other processors. These mechanisms enable the capabilitiesof the present invention in a peer-to-peer network mode of operation.The following remote mechanisms are described:

1. Locate True File;

2. Reserve True File;

3. Request True File;

4. Retire True File;

5. Cancel Reservation;

6. Acquire True File;

7. Lock Cache;

8. Update Cache; and

9. Check Expiration Date.

Background mechanisms are intended to run occasionally and at a lowpriority. These provide automated management capabilities with respectto the present invention. The following background mechanisms aredescribed:

1. Mirror True File;

2. Groom Region;

3. Check for Expired Links; and

4. Verify Region; and

5. Groom Source List.

Extended mechanisms run within application programs over the operatingsystem. These mechanisms provide solutions to specific problems andapplications. The following extended mechanisms are described:

1. Inventory Existing Directory;

2. Inventory Removable, Read-only Files;

3. Synchronize directories;

4. Publish Region;

5. Retire Directory;

6. Realize Directory at location;

7. Verify True File;

8. Track for accounting purposes; and

9. Track for licensing purposes.

The file system herein described maintains sufficient information toprovide a variety of mechanisms not ordinarily offered by an operatingsystem, some of which are listed and described here. Various processingperformed by this embodiment of the present invention will now bedescribed in greater detail.

In some embodiments, some files 120 in a data processing system 100 donot have True Names because they have been recently received or createdor modified, and thus their True Names have not yet been computed. Afile that does not yet have a True Name is called a scratch file. Theprocess of assigning a True Name to a file is referred to asassimilation, and is described later. Note that a scratch file may havea user provided name.

Some of the processing performed by the present invention can take placein a background mode or on a delayed or as-needed basis. This backgroundprocessing is used to determine information that is not immediatelyrequired by the system or which may never be required. As an example, insome cases a scratch file is being changed at a rate greater than therate at which it is useful to determine its True Name. In these cases,determining the True Name of the file can be postponed or performed inthe background.

Data Structures

The following data structures, stored in memory 110 of one of moreprocessors 102 are used to implement the mechanisms described herein.The data structures can be local to each processor 102 of the system100, or they can reside on only some of the processors 102.

The data structures described are assumed to reside on individual peerprocessors 102 in the data processing system 100. However, they can alsobe shared by placing them on a remote, shared file server (for instance,in a local area network of machines). In order to accommodate sharingdata structures, it is necessary that the processors accessing theshared database use the appropriate locking techniques to ensure thatchanges to the shared database do not interfere with one another but areappropriately serialized. These locking techniques are well understoodby ordinarily skilled programmers of distributed applications.

It is sometimes desirable to allow some regions to be local to aparticular processor 102 and other regions to be shared among processors102. (Recall that a region is a unit of file system management andcontrol consisting of a given directory identified by the pathname ofthe directory.) In the case of local and shared regions, there would beboth local and shared versions of each data structure. Simple changes tothe processes described below must be made to ensure that appropriatedata structures are selected for a given operation.

The local directory extensions (LDE) table 124 is a data structure whichprovides information about files 120 and directories 118 in the dataprocessing system 100. The local directory extensions table 124 isindexed by a pathname or contextual name (that is, a user provided name)of a file and includes the True Name for most files. The information inlocal directory extension table 124 is in addition to that provided bythe native file system of the operating system.

The True File registry (TFR) 126 is a data store for listing actual dataitems which have True Names, both files 120 and segments 122. When suchdata items occur in the True File registry 126 they are known as TrueFiles. True Files are identified in True File registry 126 by their TrueNames or identities. The table True File registry 126 also storeslocation, dependency, and migration information about True Files.

The region table (RT) 128 defines areas in the network storage which areto be managed separately. Region table 128 defines the rules for accessto and migration of files 120 among various regions with the local filesystem 116 and remote peer file systems.

The source table (ST) 130 is a list of the sources of True Files otherthan the current True File registry 126. The source table 130 includesremovable volumes and remote processors.

The audit file (AF) 132 is a list of records indicating changes to bemade in local or remote files, these changes to be processed inbackground.

The accounting log (AL) 134 is a log of file transactions used to createaccounting information in a manner which preserves the identity of filesbeing tracked independent of their name or location.

The license table (LT) 136 is a table identifying files, which may onlybe used by licensed users, in a manner independent of their name orlocation, and the users licensed to use them.

Detailed Descriptions of the Data Structures

The following table summarizes the fields of an local directoryextensions table entry, as illustrated by record 138 in FIG. 3. FieldDescription Region ID identifies the region in which this file iscontained. Pathname the user provided name or contextual name of thefile or directory, relative to the region in which it occurs. True Namethe computed True Name or identity of the file or directory. This TrueName is not always up to date, and it is set to a special value when afile is modified and is later recomputed in the background. Typeindicates whether the file is a data file or a directory. Scratch Filethe physical location of the file in the file system, when no True IDName has been calculated for the file. As noted above, such a file iscalled a scratch file. Time of last the last access time to this file.If this file is a directory, this is the access last access time to anyfile in the directory. Time of last the time of last change of thisfile. If this file is a directory, this is modification the lastmodification time of any file in the directory. Safe flag indicates thatthis file (and, if this file is a directory, all of its subordinatefiles) have been backed up on some other system, and it is thereforesafe to remove them. Lock flag indicates whether a file is locked, thatis, it is being modified by the local processor or a remote processor.Only one processor may modify a file at a time. Size the full size ofthis directory (including all subordinate files), if all files in itwere fully expanded and duplicated. For a file that is not a directorythis is the size of the actual True File. Owner the identity of the userwho owns this file, for accounting and license tracking purposes.

Each record of the True File registry 126 has the fields shown in theTrue File registry record 140 in FIG. 4. The True File registry 126consists of the database described in the table below as well as theactual True Files identified by the True File IDs below. FieldDescription True Name computed True Name or identity of the file.Compressed compressed version of the True File may be stored instead of,or in File ID addition to, an uncompressed version. This field providesthe identity of the actual representation of the compressed version ofthe file. Grooming tentative count of how many references have beenselected for delete count deletion during a grooming operation. Time oflast most recent date and time the content of this file was accessed.access Expiration date and time after which this file may be deleted bythis server. Dependent processor IDs of other processors which containreferences to this processors True File. Source IDs source ID(s) of zeroor more sources from which this file or data item may be retrieved. TrueFile ID identity or disk location of the actual physical representationof the file or file segment. It is sufficient to use a filename in theregistration directory of the underlying operating system. The True FileID is absent if the actual file is not currently present at the currentlocation. Use count number of other records on this processor whichidentify this True File.

A region table 128, specified by a directory pathname, records storagepolicies which allow files in the file system to be stored, accessed andmigrated in different ways. Storage policies are programmed in aconfigurable way using a set of rules described below.

Each region table record 142 of region table 128 includes the fieldsdescribed in the following table (with reference to FIG. 5): FieldDescription Region ID internally used identifier for this region. Regionfile file system on the local processor of which this region is a part.system Region a pathname relative to the region file system whichdefines the pathname location of this region. The region consists of allfiles and directories subordinate to this pathname, except those in aregion subordinate to this region. Mirror zero or more identifiers ofprocessors which are to keep mirror or processor(s) archival copies ofall files in the current region. Multiple mirror processors can bedefined to form a mirror group. Mirror number of copies of each file inthis region that should be retained duplication in a mirror group. countRegion specifies whether this region is local to a single processor 102,status shared by several processors 102 (if, for instance, it resides ona shared file server), or managed by a remote processor. Policy themigration policy to apply to this region. A single region mightparticipate in several policies. The policies are as follows (parametersin brackets are specified as part of the policy): region is a cachedversion from [processor ID]; region is a member of a mirror set definedby [processor ID], region is to be archived on [processor ID]. region isto be backed up locally, by placing new copies in [region ID]. region isread only and may not be changed. region is published and expires on[date]. Files in this region should be compressed.

A source table 130 identifies a source location for True Files. Thesource table 130 is also used to identify client processors makingreservations on the current processor. Each source record 144 of thesource table 130 includes the fields summarized in the following table,with reference to FIG. 6: Field Description source ID internalidentifier used to identify a particular source. source type type ofsource location: Removable Storage Volume Local Region Cache ServerMirror Group Server Cooperative Server Publishing Server Client sourceincludes information about the rights of this processor, such as rightswhether it can ask the local processor to store data items for it.source measurement of the bandwidth, cost, and reliability of theavailability connection to this source of True Files. The availabilityis used to select from among several possible sources. sourceinformation on how the local processor is to access the source. Thislocation may be, for example, the name of a removable storage volume, orthe processor ID and region path of a region on a remote processor.

The audit file 132 is a table of events ordered by timestamp, eachrecord 146 in audit file 132 including the fields summarized in thefollowing table (with reference to FIG. 7): Field Description Originalpath of the file in question. Name Operation whether the file wascreated, read, written, copied or deleted. Type specifies whether thesource is a file or a directory. Processor ID of the remote processorgenerating this event ID (if not local). Timestamp time and date filewas closed (required only for accessed/modified files). Pathname Name ofthe file (required only for rename). True Name computed True Name of thefile. This is used by remote systems to mirror changes to the directoryand is filled in during background processing.

Each record 148 of the accounting log 134 records an event which maylater be used to provide information for billing mechanisms. Eachaccounting log entry record 148 includes at least the informationsummarized in the following table, with reference to FIG. 8: FieldDescription date of entry date and time of this log entry. type of entryEntry types include create file, delete file, and transmit file. TrueName True Name of data item in question. owner identity of the userresponsible for this action.

Each record 150 of the license table 136 records a relationship betweena licensable data item and the user licensed to have access to it. Eachlicense table record 150 includes the information summarized in thefollowing table, with reference to FIG. 9: Field Description True NameTrue Name of a data item subject to license validation. licenseeidentity of a user authorized to have access to this object.

Various other data structures are employed on some or all of theprocessors 102 in the data processing system 100. Each processor 102 hasa global freeze lock (GFL) 152 (FIG. 1), which is used to preventsynchronization errors when a directory is frozen or copied. Anyprocessor 102 may include a special archive directory (SAD) 154 intowhich directories may be copied for the purposes of archival. Anyprocessor 102 may include a special media directory (SMD) 156, intowhich the directories of removable volumes are stored to form a mediainventory. Each processor has a grooming lock 158, which is set during agrooming operation. During this period the grooming delete count of TrueFile registry entries 140 is active, and no True Files should be deleteduntil grooming is complete. While grooming is in effect, groominginformation includes a table of pathnames selected for deletion, andkeeps track of the amount of space that would be freed if all of thefiles were deleted.

Primitive Mechanisms

The first of the mechanisms provided by the present invention, primitivemechanisms, are now described. The mechanisms described here depend onunderlying data management mechanisms to create, copy, read, and deletedata items in the True File registry 126, as identified by a True FileID. This support may be provided by an underlying operating system ordisk storage manager.

The following primitive mechanisms are described:

1. Calculate True Name;

2. Assimilate Data Item;

3. True File;

4. Get True Name from Path;

5. Link Path to True Name;

6. Realize True File from Location;

7. Locate Remote File;

8. Make True File Local;

9. Create Scratch File;

10. Freeze Directory;

11. Expand Frozen Directory;

12. Delete True File;

13. Process Audit File Entry;

14. Begin Grooming;

15. Select For Removal; and

16. End Grooming.

1. Calculate True Name

A True Name is computed using a function, MD, which reduces a data blockB of arbitrary length to a relatively small, fixed size identifier, theTrue Name of the data block, such that the True Name of the data blockis virtually guaranteed to represent the data block B and only datablock B.

The function MD must have the following properties:

-   -   1. The domain of the function MD is the set of all data items.        The range of the function MD is the set of True Names.    -   2. The function MD must take a data item of arbitrary length and        reduce it to an integer value in the range 0 to N−1, where N is        the cardinality of the set of True Names. That is, for an        arbitrary length data block B, 0≦MD(B)<N.    -   3. The results of MD(B) must be evenly and randomly distributed        over the range of N, in such a way that simple or regular        changes to B are virtually guaranteed to produce a different        value of MD(B).    -   4. It must be computationally difficult to find a different        value B′ such that MD(B)=MD(B′).    -   5. The function MD(B) must be efficiently computed.

A family of functions with the above properties are the so-calledmessage digest functions, which are used in digital security systems astechniques for authentification of data. These functions (or algorithms)include MD4, MD5, and SHA.

In the presently preferred embodiments, either MD5 or SHA is employed asthe basis for the computation of True Names. Whichever of these twomessage digest functions is employed, that same function must beemployed on a system-wide basis.

It is impossible to define a function having a unique output for eachpossible input when the number of elements in the range of the functionis smaller than the number of elements in its domain. However, a crucialobservation is that the actual data items that will be encountered inthe operation of any system embodying this invention form a very sparsesubset of all the possible inputs.

A colliding set of data items is defined as a set wherein, for one ormore pairs x and y in the set, MD(x)=MD(y). Since a function conformingto the requirements for MD must evenly and randomly distribute itsoutputs, it is possible, by making the range of the function largeenough, to make the probability arbitrarily small that actual inputsencountered in the operation of an embodiment of this invention willform a colliding set.

To roughly quantify the probability of a collision, assume that thereare no more than 2³⁰ storage devices in the world, and that each storagedevice has an average of at most 2²⁰ different data items. Then thereare at most 2⁵⁰ data items in the world. If the outputs of MD rangebetween 0 and 2²⁸, it can be demonstrated that the probability of acollision is approximately 1 in 2²⁹. Details on the derivation of theseprobability values are found, for example, in P. Flajolet and A. M.Odlyzko, “Random Mapping Statistics,” Lecture Notes in Computer Science434: Advances in Cryptology—Eurocrypt '89 Proceedings, Springer-Verlag,pp. 329-354.

Note that for some less preferred embodiments of the present invention,lower probabilities of uniqueness may be acceptable, depending on thetypes of applications and mechanisms used. In some embodiments it mayalso be useful to have more than one level of True Names, with some ofthe True Names having different degrees of uniqueness. If such a schemeis implemented, it is necessary to ensure that less unique True Namesare not propagated in the system.

While the invention is described herein using only the True Name of adata item as the identifier for the data item, other preferredembodiments use tagged, typed, categorized or classified data items anduse a combination of both the True Name and the tag, type, category orclass of the data item as an identifier. Examples of suchcategorizations are files, directories, and segments; executable filesand data files, and the like. Examples of classes are classes of objectsin an object-oriented system. In such a system, a lower degree of TrueName uniqueness is acceptable over the entire universe of data items, aslong as sufficient uniqueness. is provided per category of data items.This is because the tags provide an additional level of uniqueness.

A mechanism for calculating a True Name given a data item is nowdescribed, with reference to FIGS. 10(a) and 10(b).

A simple data item is a data item whose size is less than a particulargiven size (which must be defined in each particular implementation ofthe invention). To determine the True Name of a simple data item, withreference to FIG. 10(a), first compute the MD function (described above)on the given simple data item (Step S212). Then append to the resulting128 bits, the byte length modulo 32 of the data item (Step S214). Theresulting 160-bit value is the True Name of the simple data item.

A compound data item is one whose size is greater than the particulargiven size of a simple data item. To determine the True Name of anarbitrary (simple or compound) data item, with reference to FIG. 10(b),first determine if the data item is a simple or a compound data item(Step S216). If the data item is a simple data item, then compute itsTrue Name in step S218 (using steps S212 and S214 described above),otherwise partition the data item into segments (Step S220) andassimilate each segment (Step S222) (the primitive mechanism, Assimilatea Data Item, is described below), computing the True Name of thesegment. Then create an indirect block consisting of the computedsegment True Names (Step S224). An indirect block is a data item whichconsists of the sequence of True Names of the segments. Then, in stepS226, assimilate the indirect block and compute its True Name. Finally,replace the final thirty-two (32) bits of the resulting True Name (thatis, the length of the indirect block) by the length modulo 32 of thecompound data item (Step S228). The result is the True Name of thecompound data item.

Note that the compound data item may be so large that the indirect blockof segment True Names is itself a compound data item. In this case themechanism is invoked recursively until only simple data items are beingprocessed.

Both the use of segments and the attachment of a length to the True Nameare not strictly required in a system using the present invention, butare currently considered desirable features in the preferred embodiment.

2. Assimilate Data Item

A mechanism for assimilating a data item (scratch file or segment) intoa file system, given the scratch file ID of the data item, is nowdescribed with reference to FIG. 11. The purpose of this mechanism is toadd a given data item to the True File registry 126. If the data itemalready exists in the True File registry 126, this will be discoveredand used during this process, and the duplicate will be eliminated.

Thereby the system stores at most one copy of any data item or file bycontent, even when multiple names refer to the same content.

First, determine the True Name of the data item corresponding to thegiven scratch File ID using the Calculate True Name primitive mechanism(Step S230). Next, look for an entry for the True Name in the True Fileregistry 126 (Step S232) and determine whether a True Name entry, record140, exists in the True File registry 126. If the entry record includesa corresponding True File ID or compressed File ID (Step S237), deletethe file with the scratch File ID (Step S238). Otherwise store the givenTrue File ID in the entry record (step S239).

If it is determined (in step S232) that no True Name entry exists in theTrue File registry 126, then, in Step S236, create a new entry in theTrue File registry 126 for this True Name. Set the True Name of theentry to the calculated True Name, set the use count for the new entryto one, store the given True File ID in the entry and set the otherfields of the entry as appropriate.

Because this procedure may take some time to compute, it is intended torun in background after a file has ceased to change. In the meantime,the file is considered an unassimilated scratch file.

3. True File

The True File process is invoked when processing the audit file 132,some time after a True File has been assimilated (using the AssimilateData Item primitive mechanism). Given a local directory extensions tableentry record 138 in the local directory extensions table 124, the TrueFile process can provide the following steps (with reference to FIG.12), depending on how the local processor is configured:

First, in step S238, examine the local directory extensions table entryrecord 138 to determine whether the file is locked by a cache server. Ifthe file is locked, then add the ID of the cache server to the dependentprocessor list of the True File registry table 126, and then send amessage to the cache server to update the cache of the current processorusing the Update Cache remote mechanism (Step 242).

If desired, compress the True File (Step S246), and, if desired, mirrorthe True File using the Mirror True File background mechanism (StepS248).

4. Get True Name from Path

The True Name of a file can be used to identify a file by contents, toconfirm that a file matches its original contents, or to compare twofiles. The mechanism to get a True Name given the pathname of a file isnow described with reference to FIG. 13.

First, search the local directory extensions table 124 for the entryrecord 138 with the given pathname (Step S250). If the pathname is notfound, this process fails and no True Name corresponding to the givenpathname exists. Next, determine whether the local directory extensionstable entry record 138 includes a True Name (Step S252), and if so, themechanism's task is complete. Otherwise, determine whether the localdirectory extensions table entry record 138 identifies a directory (StepS254), and if so, freeze the directory (Step S256) (the primitivemechanism Freeze Directory is described below).

Otherwise, in step S258, assimilate the file (using the Assimilate DataItem primitive mechanism) defined by the File ID field to generate itsTrue Name and store its True Name in the local directory extensionsentry record. Then return the True Name identified by the localdirectory extensions table 124.

5. Link Path to True Name

The mechanism to link a path to a True Name provides a way of creating anew directory entry record identifying an existing, assimilated file.This basic process may be used to copy, move, and rename files without aneed to copy their contents. The mechanism to link a path to a True Nameis now described with reference to FIG. 14.

First, if desired, confirm that the True Name exists locally bysearching for it in the True Name registry or local directory extensionstable 135 (Step S260). Most uses of this mechanism will require thisform of validation. Next, search for the path in the local directoryextensions table 135 (Step S262). Confirm that the directory containingthe file named in the path already exists (Step S264). If the named fileitself exists, delete the File using the Delete True File operatingsystem mechanism (see below) (Step S268).

Then, create an entry record in the local directory extensions with thespecified path (Step S270) and update the entry record and other datastructures as follows: fill in the True Name field of the entry with thespecified True Name; increment the use count for the True File registryentry record 140 of the corresponding True Name; note whether the entryis a directory by reading the True File to see if it contains a tag(magic number) indicating that it represents a frozen directory (seealso the description of the Freeze Directory primitive mechanismregarding the tag); and compute and set the other fields of the localdirectory extensions appropriately. For instance, search the regiontable 128 to identify the region of the path, and set the time of lastaccess and time of last modification to the current time.

6. Realize True File from Location

This mechanism is used to try to make a local copy of a True File, givenits True Name and the name of a source location (processor or media)that may contain the True File. This mechanism is now described withreference to FIG. 15.

First, in step S272, determine whether the location specified is aprocessor. If it is determined that the location specified is aprocessor, then send a Request True File message (using the Request TrueFile remote mechanism) to the remote processor and wait for a response(Step S274). If a negative response is received or no response isreceived after a timeout period, this mechanism fails. If a positiveresponse is received, enter the True File returned in the True Fileregistry 126 (Step S276). (If the file received was compressed, enterthe True File ID in the compressed File ID field.)

If, on the other hand, it is determined in step S272 that the locationspecified is not a processor, then, if necessary, request the user oroperator to mount the indicated volume (Step S278). Then (Step S280)find the indicated file on the given volume and assimilate the fileusing the Assimilate Data Item primitive mechanism. If the volume doesnot contain a True File registry 126, search the media inventory to findthe path of the file on the volume. If no such file can be found, thismechanism fails.

At this point, whether or not the location is determined (in step S272)to be a processor, if desired, verify the True File (in step S282).

7. Locate Remote File

This mechanism allows a processor to locate a file or data item from aremote source of True Files, when a specific source is unknown orunavailable. A client processor system may ask one of several or manysources whether it can supply a data object with a given True Name. Thesteps to perform this mechanism are as follows (with reference to FIGS.16(a) and 16(b)).

The client processor 102 uses the source table 145 to select one or moresource processors (Step S284). If no source processor can be found, themechanism fails. Next, the client processor 102 broadcasts to theselected sources a request to locate the file with the given True Nameusing the Locate True File remote mechanism (Step S286). The request tolocate may be augmented by asking to propagate this request to distantservers. The client processor then waits for one or more servers torespond positively (Step S288). After all servers respond negatively, orafter a timeout period with no positive response, the mechanism repeatsselection (Step S284) to attempt to identify alternative sources. If anyselected source processor responds, its processor ID is the result ofthis mechanism. Store the processor ID in the source field of the TrueFile registry entry record 140 of the given True Name (Step S290).

If the source location of the True Name is a different processor ormedium than the destination (Step S290 a), perform the following steps:

-   -   (i) Look up the True File registry entry record 140 for the        corresponding True Name, and add the source location ID to the        list of sources for the True Name (Step S290 b); and    -   (ii) If the source is a publishing system, determine the        expiration date on the publishing system for the True Name and        add that to the list of sources. If the source is not a        publishing system, send a message to reserve the True File on        the source processor (Step S290 c).

Source selection in step S284 may be based on optimizations involvinggeneral availability of the source, access time, bandwidth, andtransmission cost, and ignoring previously selected processors which didnot respond in step S288.

8. Make True File Local

This mechanism is used when a True Name is known and a locallyaccessible copy of the corresponding file or data item is required. Thismechanism makes it possible to actually read the data in a True File.The mechanism takes a True Name and returns when there is a local,accessible copy of the True File in the True File registry 126. Thismechanism is described here with reference to the flow chart of FIGS.17(a) and 17(b).

First, look in the True File registry 126 for a True File entry record140 for the corresponding True Name (Step S292). If no such entry isfound this mechanism fails. If there is already a True File ID for theentry (Step S294), this mechanism's task is complete. If there is acompressed file ID for the entry (Step S296), decompress the filecorresponding to the file ID (Step S298) and store the decompressed fileID in the entry (Step S300). This mechanism is then complete.

If there is no True File ID for the entry (Step S294) and there is nocompressed file ID for the entry (Step S296), then continue searchingfor the requested file. At this time it may be necessary to notify theuser that the system is searching for the requested file.

If there are one or more source IDs, then select an order in which toattempt to realize the source ID (Step S304). The order may be based onoptimizations involving general availability of the source, access time,bandwidth, and transmission cost. For each source in the order chosen,realize the True File from the source location (using the Realize TrueFile from Location primitive mechanism), until the True File is realized(Step S306). If it is realized, continue with step S294. If no knownsource can realize the True File, use the Locate Remote File primitivemechanism to attempt to find the True File (Step S308). If thissucceeds, realize the True File from the identified source location andcontinue with step S296.

9. Create Scratch File

A scratch copy of a file is required when a file is being created or isabout to be modified. The scratch copy is stored in the file system ofthe underlying operating system. The scratch copy is eventuallyassimilated when the audit file record entry 146 is processed by theProcess Audit File Entry primitive mechanism. This Create Scratch Filemechanism requires a local directory extensions table entry record 138.When it succeeds, the local directory extensions table entry record 138contains the scratch file ID of a scratch file that is not contained inthe True File registry 126 and that may be modified. This mechanism isnow described with reference to FIGS. 18(a) and 18(b).

First determine whether the scratch file should be a copy of theexisting True File (Step S310). If so, continue with step S312.Otherwise, determine whether the local directory extensions table entryrecord 138 identifies an existing True File (Step S316), and if so,delete the True File using the Delete True File primitive mechanism(Step S318). Then create a new, empty scratch file and store its scratchfile ID in the local directory extensions table entry record 138 (stepS320). This mechanism is then complete.

If the local directory extensions table entry record 138 identifies ascratch file ID (Step S312), then the entry already has a scratch file,so this mechanism succeeds.

If the local directory extensions table entry record 138 identifies aTrue File (S316), and there is no True File ID for the True File (S312),then make the True File local using the Make True File Local primitivemechanism (Step S322). If there is still no True File ID, this mechanismfails.

There is now a local True File for this file. If the use count in thecorresponding True File registry entry record 140 is one (Step S326),save the True File ID in the scratch file ID of the local directoryextensions table entry record 138, and remove the True File registryentry record 140 (Step S328). (This step makes the True File into ascratch file.) This mechanism's task is complete.

Otherwise, if the use count in the corresponding True File registryentry record 140 is not one (in step S326), copy the file with the givenTrue File ID to a new scratch file, using the Read File OS mechanism andstore its file ID in the local directory extensions table entry record138 (Step S330), and reduce the use count for the True File by one. Ifthere is insufficient space to make a copy, this mechanism fails.

10. Freeze Directory

This mechanism freezes a directory in order to calculate its True Name.Since the True Name of a directory is a function of the files within thedirectory, they must not change during the computation of the True Nameof the directory. This mechanism requires the pathname of a directory tofreeze. This mechanism is described with reference to FIGS. 19(a) and19(b).

In step S332, add one to the global freeze lock. Then search the localdirectory extensions table 124 to find each subordinate data file anddirectory of the given directory, and freeze each subordinate directoryfound using the Freeze Directory primitive mechanism (Step S334).Assimilate each unassimilated data file in the directory using theAssimilate Data Item primitive mechanism (Step S336). Then create a dataitem which begins with a tag or marker (a “magic number”) being a uniquedata item indicating that this data item is a frozen directory (StepS337). Then list the file name and True Name for each file in thecurrent directory (Step S338). Record any additional informationrequired, such as the type, time of last access and modification, andsize (Step S340). Next, in step S342, using the Assimilate Data Itemprimitive mechanism, assimilate the data item created in step S338. Theresulting True Name is the True Name of the frozen directory. Finally,subtract one from the global freeze lock (Step S344).

11. Expand Frozen Directory

This mechanism expands a frozen directory in a given location. Itrequires a given pathname into which to expand the directory, and theTrue Name of the directory and is described with reference to FIG. 20.

First, in step S346, make the True File with the given True Name localusing the Make True File Local primitive mechanism. Then read eachdirectory entry in the local file created in step S346 (Step S348). Foreach such directory entry, do the following:

Create a full pathname using the given pathname and the file name of theentry (Step S350); and

link the created path to the True Name (Step S352) using the Link Pathto True Name primitive mechanism.

12. Delete True File

This mechanism deletes a reference to a True Name. The underlying TrueFile is not removed from the True File registry 126 unless there are noadditional references to the file. With reference to FIG. 21, thismechanism is performed as follows:

If the global freeze lock is on, wait until the global freeze lock isturned off (Step S354). This prevents deleting a True File while adirectory which might refer to it is being frozen. Next, find the TrueFile registry entry record 140 given the True Name (Step S356). If thereference count field of the True File registry 126 is greater thanzero, subtract one from the reference count field (Step S358). If it isdetermined (in step S360) that the reference count field of the TrueFile registry entry record 140 is zero, and if there are no dependentsystems listed in the True File registry entry record 140, then performthe following steps:

(i) If the True File is a simple data item, then delete the True File,otherwise,

(ii) (the True File is a compound data item) for each True Name in thedata item, recursively delete the True File corresponding to the TrueName (Step S362).

(iii) Remove the file indicated by the True File ID and compressed fileID from the True File registry 126, and remove the True File registryentry record 140 (Step S364).

13. Process Audit File Entry

This mechanism performs tasks which are required to maintain informationin the local directory extensions table 124 and True File registry 126,but which can be delayed while the processor is busy doing moretime-critical tasks. Entries 142 in the audit file 132 should beprocessed at a background priority as long as there are entries to beprocessed. With reference to FIG. 22, the steps for processing an entryare as follows:

Determine the operation in the entry 142 currently being processed (StepS365). If the operation indicates that a file was created or written(Step S366), then assimilate the file using the Assimilate Data Itemprimitive mechanism (Step S368), use the True File primitive mechanismto do additional desired processing (such as cache update, compression,and mirroring) (Step S369), and record the newly computed True Name forthe file in the audit file record entry (Step S370).

Otherwise, if the entry being processed indicates that a compound dataitem or directory was copied (or deleted) (Step S376), then for eachcomponent True Name in the compound data item or directory, add (orsubtract) one to the use count of the True File registry entry record140 corresponding to the component True Name (Step S378).

In all cases, for each parent directory of the given file, update thesize, time of last access, and time of last modification, according tothe operation in the audit record (Step S379).

Note that the audit record is not removed after processing, but isretained for some reasonable period so that it may be used by theSynchronize Directory extended mechanism to allow a disconnected remoteprocessor to update its representation of the local system.

14. Begin Grooming

This mechanism makes it possible to select a set of files for removaland determine the overall amount of space to be recovered. Withreference to FIG. 23, first verify that the global grooming lock iscurrently unlocked (Step S382). Then set the global grooming lock, setthe total amount of space freed during grooming to zero and empty thelist of files selected for deletion (Step S384). For each True File inthe True File registry 126, set the delete count to zero (Step S386).

15. Select For Removal

This grooming mechanism tentatively selects a pathname to allow itscorresponding True File to be removed. With reference to FIG. 24, firstfind the local directory extensions table entry record 138 correspondingto the given pathname (Step S388). Then find the True File registryentry record 140 corresponding to the True File name in the localdirectory extensions table entry record 138 (Step S390). Add one to thegrooming delete count in the True File registry entry record 140 and addthe pathname to a list of files selected for deletion (Step S392). Ifthe grooming delete count of the True File registry entry record 140 isequal to the use count of the True File registry entry record 140, andif the there are no entries in the dependency list of the True Fileregistry entry record 140, then add the size of the file indicated bythe True File ID and or compressed file ID to the total amount of spacefreed during grooming (Step S394).

16. End Grooming

This grooming mechanism ends the grooming phase and removes all filesselected for removal. With reference to FIG. 25, for each file in thelist of files selected for deletion, delete the file (Step S396) andthen unlock the global grooming lock (Step S398).

Operating System Mechanisms

The next of the mechanisms provided by the present invention, operatingsystem mechanisms, are now described.

The following operating system mechanisms are described:

1. Open File;

2. Close File;

3. Read File;

4. Write File;

5. Delete File or Directory;

6. Copy File or Directory;

7. Move File or Directory;

8. Get File Status; and

9. Get Files in Directory.

1. Open File

A mechanism to open a file is described with reference to FIGS. 26(a)and 26(b). This mechanism is given as input a pathname and the type ofaccess required for the file (for example, read, write, read/write,create, etc.) and produces either the File ID of the file to be openedor an indication that no file should be opened. The local directoryextensions table record 138 and region table record 142 associated withthe opened file are associated with the open file for later use in otherprocessing functions which refer to the file, such as read, write, andclose.

First, determine whether or not the named file exists locally byexamining the local directory extensions table 124 to determine whetherthere is an entry corresponding to the given pathname (Step S400). If itis determined that the file name does not exist locally, then, using theaccess type, determine whether or not the file is being created by thisopening process (Step S402). If the file is not being created, prohibitthe open (Step S404). If the file is being created, create a zero-lengthscratch file using an entry in local directory extensions table 124 andproduce the scratch file ID of this scratch file as the result (StepS406).

If, on the other hand, it is determined in step S400 that the file namedoes exist locally, then determine the region in which the file islocated by searching the region table 128 to find the record 142 withthe longest region path which is a prefix of the file pathname (StepS408). This record identifies the region of the specified file.

Next, determine using the access type, whether the file is being openedfor writing or whether it is being opened only for reading (Step S410).If the file is being opened for reading only, then, if the file is ascratch file (Step S419), return the scratch File ID of the file (StepS424). Otherwise get the True Name from the local directory extensionstable 124 and make a local version of the True File associated with theTrue Name using the Make True File Local primitive mechanism, and thenreturn the True File ID associated with the True Name (Step S420).

If the file is not being opened for reading only (Step S410), then, ifit is determined by inspecting the region table entry record 142 thatthe file is in a read-only directory (Step S416), then prohibit theopening (Step S422).

If it is determined by inspecting the region table 128 that the file isin a cached region (Step S423), then send a Lock Cache message to thecorresponding cache server, and wait for a return message (Step S418).If the return message says the file is already locked, prohibit theopening.

If the access type indicates that the file being modified is beingrewritten completely (Step S419), so that the original data will not berequired, then Delete the File using the Delete File OS mechanism (StepS421) and perform step S406. Otherwise, make a scratch copy of the file(Step S417) and produce the scratch file ID of the scratch file as theresult (Step S424).

2. Close File

This mechanism takes as input the local directory extensions table entryrecord 138 of an open file and the data maintained for the open file. Toclose a file, add an entry to the audit file indicating the time andoperation (create, read or write). The audit file processing (using theProcess Audit File Entry primitive mechanism) will take care ofassimilating the file and thereby updating the other records.

3. Read File

To read a file, a program must provide the offset and length of the datato be read, and the location of a buffer into which to copy the dataread.

The file to be read from is identified by an open file descriptor whichincludes a File ID as computed by the Open File operating systemmechanism defined above. The File ID may identify either a scratch fileor a True File (or True File segment). If the File ID identifies a TrueFile, it may be either a simple or a compound True File. Reading a fileis accomplished by the following steps:

In the case where the File ID identifies a scratch file or a simple TrueFile, use the read capabilities of the underlying operating system.

In the case where the File ID identifies a compound file, break the readoperation into one or more read operations on component segments asfollows:

A. Identify the segment(s) to be read by dividing the specified fileoffset and length each by the fixed size of a segment (a systemdependent parameter), to determine the segment number and number ofsegments that must be read.

B. For each segment number computed above, do the following:

-   -   i. Read the compound True File index block to determine the True        Name of the segment to be read.    -   ii. Use the Realize True File from Location primitive mechanism        to make the True File segment available locally. (If that        mechanism fails, the Read File mechanism fails).    -   iii. Determine the File ID of the True File specified by the        True Name corresponding to this segment.    -   iv. Use the Read File mechanism (recursively) to read from this        segment into the corresponding location in the specified buffer.        4. Write File

File writing uses the file ID and data management capabilities of theunderlying operating system. File access (Make File Local describedabove) can be deferred until the first read or write.

5. Delete File or Directory

The process of deleting a file, for a given pathname, is described herewith reference to FIGS. 27(a) and 27(b).

First, determine the local directory extensions table entry record 138and region table entry record 142 for the file (Step S422). If the filehas no local directory extensions table entry record 138 or is locked oris in a read-only region, prohibit the deletion.

Identify the corresponding True File given the True Name of the filebeing deleted using the True File registry 126 (Step S424). If the filehas no True Name, (Step S426) then delete the scratch copy of the filebased on its scratch file ID in the local directory extensions table 124(Step S427), and continue with step S428.

If the file has a True Name and the True File's use count is one (StepS429), then delete the True File (Step S430), and continue with stepS428.

If the file has a True Name and the True File's use count is greaterthan one, reduce its use count by one (Step S431). Then proceed withstep S428.

In Step S428, delete the local directory extensions table entry record,and add an entry to the audit file 132 indicating the time and theoperation performed (delete).

6. Copy File or Directory

A mechanism is provided to copy a file or directory given a source anddestination processor and pathname. The Copy File mechanism does notactually copy the data in the file, only the True Name of the file. Thismechanism is performed as follows:

(A) Given the source path, get the True Name from the path. If this stepfails, the mechanism fails.

(B) Given the True Name and the destination path, link the destinationpath to the True Name.

(C) If the source and destination processors have different True Fileregistries, find (or, if necessary, create) an entry for the True Namein the True File registry table 126 of the destination processor. Enterinto the source ID field of this new entry the source processoridentity.

(D) Add an entry to the audit file 132 indicating the time and operationperformed (copy).

This mechanism addresses capability of the system to avoid copying datafrom a source location to a destination location when the destinationalready has the data. In addition, because of the ability to freeze adirectory, this mechanism also addresses capability of the systemimmediately to make a copy of any collection of files, thereby tosupport an efficient version control mechanisms for groups of files.

7. Move File or Directory

A mechanism is described which moves (or renames) a file from a sourcepath to a destination path. The move operation, like the copy operation,requires no actual transfer of data, and is performed as follows:

(A) Copy the file from the source path to the destination path.

(B) If the source path is different from the destination path, deletethe source path.

8. Get File Status

This mechanism takes a file pathname and provides information about thepathname. First the local directory extensions table entry record 138corresponding to the pathname given is found. If no such entry exists,then this mechanism fails, otherwise, gather information about the fileand its corresponding True File from the local directory extensionstable 124. The information can include any information shown in the datastructures, including the size, type, owner, True Name, sources, time oflast access, time of last modification, state (local or not, assimilatedor not, compressed or not), use count, expiration date, andreservations.

9. Get Files in Directory

This mechanism enumerates the files in a directory. It is used(implicitly) whenever it is necessary to determine whether a file exists(is present) in a directory. For instance, it is implicitly used in theOpen File, Delete File, Copy File or Directory, and Move File operatingsystem mechanisms, because the files operated on are referred to bypathnames containing directory names. The mechanism works as follows:

The local directory extensions table 124 is searched for an entry 138with the given directory pathname. If no such entry is found, or if theentry found is not a directory, then this mechanism fails.

If there is a corresponding True File field in the local directoryextensions table record, then it is assumed that the True Filerepresents a frozen directory. The Expand Frozen Directory primitivemechanism is used to expand the existing True File into directoryentries in the local directory extensions table.

Finally, the local directory extensions table 124 is again searched,this time to find each directory subordinate to the given directory. Thenames found are provided as the result.

Remote Mechanisms

The remote mechanisms provided by the present invention are nowdescribed. Recall that remote mechanisms are used by the operatingsystem in responding to requests from other processors. These mechanismsenable the capabilities of the present invention in a peer-to-peernetwork mode of operation.

In a presently preferred embodiment, processors communicate with eachother using a remote procedure call (RPC) style interface, running overone of any number of communication protocols such as IPX/SPX or TCP/IP.Each peer processor which provides access to its True File registry 126or file regions, or which depends on another peer processor, provides anumber of mechanisms which can be used by its peers.

The following remote mechanisms are described:

1. Locate True File;

2. Reserve True File;

3. Request True File;

4. Retire True File;

5. Cancel Reservation;

6. Acquire True File;

7. Lock Cache;

8. Update Cache; and

9. Check Expiration Date.

1. Locate True File

This mechanism allows a remote processor to determine whether the localprocessor contains a copy of a specific True File. The mechanism beginswith a True Name and a flag indicating whether to forward requests forthis file to other servers. This mechanism is now described withreference to FIG. 28.

First determine if the True File is available locally or if there issome indication of where the True File is located (for example, in theSource IDs field). Look up the requested True Name in the True Fileregistry 126 (Step S432).

If a True File registry entry record 140 is not found for this True Name(Step S434), and the flag indicates that the request is not to beforwarded (Step S436), respond negatively (Step S438). That is, respondto the effect that the True File is not available.

One the other hand, if a True File registry entry record 140 is notfound (Step S434), and the flag indicates that the request for this TrueFile is to be forwarded (Step S436), then forward a request for thisTrue File to some other processors in the system (Step S442). If thesource table for the current processor identifies one or more publishingservers which should have a copy of this True File, then forward therequest to each of those publishing servers (Step S436).

If a True File registry entry record 140 is found for the required TrueFile (Step S434), and if the entry includes a True File ID or CompressedFile ID (Step S440), respond positively (Step S444). If the entryincludes a True File ID then this provides the identity or disk locationof the actual physical representation of the file or file segmentrequired. If the entry include a Compressed File ID, then a compressedversion of the True File may be stored instead of, or in addition to, anuncompressed version. This field provides the identity of the actualrepresentation of the compressed version of the file.

If the True File registry entry record 140 is found (Step S434) but doesnot include a True File ID (the File ID is absent if the actual file isnot currently present at the current location) (Step S440), and if theTrue File registry entry record 140 includes one or more sourceprocessors, and if the request can be forwarded, then forward therequest for this True File to one or more of the source processors (StepS444).

2. Reserve True File

This mechanism allows a remote processor to indicate that it depends onthe local processor for access to a specific True File. It takes a TrueName as input. This mechanism is described here.

(A) Find the True File registry entry record 140 associated with thegiven True File. If no entry exists, reply negatively.

(B) If the True File registry entry record 140 does not include a TrueFile ID or compressed File ID, and if the True File registry entryrecord 140 includes no source IDs for removable storage volumes, thenthis processor does not have access to a copy of the given file. Replynegatively.

(C) Add the ID of the sending processor to the list of dependentprocessors for the True File registry entry record 140. Replypositively, with an indication of whether the reserved True File is online or off line.

3. Request True File

This mechanism allows a remote processor to request a copy of a TrueFile from the local processor. It requires a True Name and respondspositively by sending a True File back to the requesting processor. Themechanism operates as follows:

(A) Find the True File registry entry record 140 associated with thegiven True Name. If there is no such True File registry entry record140, reply negatively.

(B) Make the True File local using the Make True File Local primitivemechanism. If this mechanism fails, the Request True File mechanism alsofails.

(C) Send the local True File in either it is uncompressed or compressedform to the requesting remote processor. Note that if the True File is acompound file, the components are not sent.

(D) If the remote file is listed in the dependent process list of theTrue File registry entry record 140, remove it.

4. Retire True File

This mechanism allows a remote processor to indicate that it no longerplans to maintain a copy of a given True File. An alternate source ofthe True File can be specified, if, for instance, the True File is beingmoved from one server to another. It begins with a True Name, arequesting processor ID, and an optional alternate source. Thismechanism operates as follows:

(A) Find a True Name entry in the True File registry 126. If there is noentry for this True Name, this mechanism's task is complete.

(B) Find the requesting processor on the source list and, if it isthere, remove it.

(C) If an alternate source is provided, add it to the source list forthe True File registry entry record 140.

(D) If the source list of the True File registry entry record 140 has noitems in it, use the Locate Remote File primitive mechanism to searchfor another copy of the file. If it fails, raise a serious error.

5. Cancel Reservation

This mechanism allows a remote processor to indicate that it no longerrequires access to a True File stored on the local processor. It beginswith a True Name and a requesting processor ID and proceeds as follows:

(A) Find the True Name entry in the True File registry 126. If there isno entry for this True Name, this mechanism's task is complete.

(B) Remove the identity of the requesting processor from the list ofdependent processors, if it appears.

(C) If the list of dependent processors becomes zero and the use countis also zero, delete the True File.

6. Acquire True File

This mechanism allows a remote processor to insist that a localprocessor make a copy of a specified True File. It is used, for example,when a cache client wants to write through a new version of a file. TheAcquire True File mechanism begins with a data item and an optional TrueName for the data item and proceeds as follows:

(A) Confirm that the requesting processor has the right to require thelocal processor to acquire data items. If not, send a negative reply.

(B) Make a local copy of the data item transmitted by the remoteprocessor.

(C) Assimilate the data item into the True File registry of the localprocessor.

(D) If a True Name was provided with the file, the True Name calculationcan be avoided, or the mechanism can verify that the file receivedmatches the True Name sent.

(E) Add an entry in the dependent processor list of the true fileregistry record indicating that the requesting processor depends on thiscopy of the given True File.

(F) Send a positive reply.

7. Lock Cache

This mechanism allows a remote cache client to lock a local file so thatlocal users or other cache clients cannot change it while the remoteprocessor is using it. The mechanism begins with a pathname and proceedsas follows:

(A) Find the local directory extensions table entry record 138 of thespecified pathname. If no such entry exists, reply negatively.

(B) If an local directory extensions table entry record 138 exists andis already locked, reply negatively that the file is already locked.

(C) If an local directory extensions table entry record 138 exists andis not locked, lock the entry. Reply positively.

8. Update Cache

This mechanism allows a remote cache client to unlock a local file andupdate it with new contents. It begins with a pathname and a True Name.The file corresponding to the True Name must be accessible from theremote processor. This mechanism operates as follows:

Find the local directory extensions table entry record 138 correspondingto the given pathname. Reply negatively if no such entry exists or ifthe entry is not locked.

Link the given pathname to the given True Name using the Link Path toTrue Name primitive mechanism.

Unlock the local directory extensions table entry record 138 and returnpositively.

9. Check Expiration Date

Return current or new expiration date and possible alternative source tocaller.

Background Processes and Mechanisms

The background processes and mechanisms provided by the presentinvention are now described. Recall that background mechanisms areintended to run occasionally and at a low priority to provide automatedmanagement capabilities with respect to the present invention.

The following background mechanisms are described:

1. Mirror True File;

2. Groom Region;

3. Check for Expired Links;

4. Verify Region; and

5. Groom Source List.

1. Mirror True File

This mechanism is used to ensure that files are available in alternatelocations in mirror groups or archived on archival servers. Themechanism depends on application-specific migration/archival criteria(size, time since last access, number of copies required, number ofexisting alternative sources) which determine under what conditions afile should be moved. The Mirror True File mechanism operates asfollows, using the True File specified, perform the following steps:

(A) Count the number of available locations of the True File byinspecting the source list of the True File registry entry record 140for the True File. This step determines how many copies of the True.File are available in the system.

(B) If the True File meets the specified migration criteria, select amirror group server to which a copy of the file should be sent. Use theAcquire True File remote mechanism to copy the True File to the selectedmirror group server. Add the identity of the selected system to thesource list for the True File.

2. Groom Region

This mechanism is used to automatically free up space in a processor bydeleting data items that may be available elsewhere. The mechanismdepends on application-specific grooming criteria (for instance, a filemay be removed if there is an alternate online source for it, it has notbeen accessed in a given number of days, and it is larger than a givensize). This mechanism operates as follows:

Repeat the following steps (i) to (iii) with more aggressive groomingcriteria until sufficient space is freed or until all grooming criteriahave been exercised. Use grooming information to determine how muchspace has been freed. Recall that, while grooming is in effect, groominginformation includes a table of pathnames selected for deletion, andkeeps track of the amount of space that would be freed if all of thefiles were deleted.

(i) Begin Grooming (using the primitive mechanism).

(ii) For each pathname in the specified region, for the True Filecorresponding to the pathname, if the True File is present, has at leastone alternative source, and meets application specific grooming criteriafor the region, select the file for removal (using the primitivemechanism).

(iii) End Grooming (using the primitive mechanism).

If the region is used as a cache, no other processors are dependent onTrue Files to which it refers, and all such True Files are mirroredelsewhere. In this case, True Files can be removed with impunity. For acache region, the grooming criteria would ordinarily eliminate the leastrecently accessed True Files first. This is best done by sorting theTrue Files in the region by the most recent access time beforeperforming step (ii) above. The application specific criteria would thusbe to select for removal every True File encountered (beginning with theleast recently used) until the required amount of free space is reached.

3. Check for Expired Links

This mechanism is used to determine whether dependencies on publishedfiles should be refreshed. The following steps describe the operation ofthis mechanism:

For each pathname in the specified region, for each True Filecorresponding to the pathname, perform the following step:

If the True File registry entry record 140 corresponding to the TrueFile contains at least one source which is a publishing server, and ifthe expiration date on the dependency is past or close, then perform thefollowing steps:

(A) Determine whether the True File registry entry record contains othersources which have not expired.

(B) Check the True Name expiration of the server. If the expiration datehas been extended, or an alternate source is suggested, add the sourceto the True File registry entry record 140.

(C) If no acceptable alternate source was found in steps (A) or (B)above, make a local copy of the True File.

(D) Remove the expired source.

4. Verify Region

This mechanism can be used to ensure that the data items in the TrueFile registry 126 have not been damaged accidentally or maliciously. Theoperation of this mechanism is described by the following steps:

(A) Search the local directory extensions table 124 for each pathname inthe specified region and then perform the following steps:

-   -   (i) Get the True File name corresponding to the pathname;    -   (ii) If the True File registry entry 140 for the True File does        not have a True File ID or compressed file ID, ignore it.    -   (iii) Use the Verify True File mechanism (see extended        mechanisms below) to confirm that the True File specified is        correct.        5. Groom Source List

The source list in a True File entry should be groomed sometimes toensure there are not too many mirror or archive copies. When a file isdeleted or when a region definition or its mirror criteria are changed,it may be necessary to inspect the affected True Files to determinewhether there are too many mirror copies. This can be done with thefollowing steps:

For each affected True File,

(A) Search the local directory extensions table to find each region thatrefers to the True File.

(B) Create a set of “required sources”, initially empty.

(C) For each region found,

-   -   (a) determine the mirroring criteria for that region,    -   (b) determine which sources for the True File satisfy the        mirroring criteria, and    -   (c) add these sources to the set of required sources.

(D) For each source in the True File registry entry, if the sourceidentifies a remote processor (as opposed to removable media), and ifthe source is not a publisher, and if the source is not in the set ofrequired sources, then eliminate the source, and use the CancelReservation remote mechanism to eliminate the given processor from thelist of dependent processors recorded at the remote processor identifiedby the source.

Extended Mechanisms

The extended mechanisms provided by the present invention are nowdescribed. Recall that extended mechanisms run within applicationprograms over the operating system to provide solutions to specificproblems and applications.

The following extended mechanisms are described:

1. Inventory Existing Directory;

2. Inventory Removable, Read-only Files;

3. Synchronize Directories;

4. Publish Region;

5. Retire Directory;

6. Realize Directory at Location;

7. Verify True File;

8. Track for Accounting Purposes; and

9. Track for Licensing Purposes.

1. Inventory Existing Directory

This mechanism determines the True Names of files in an existing on-linedirectory in the underlying operating system. One purpose of thismechanism is to install True Name mechanisms in an existing file system.

An effect of such an installation is to eliminate immediately allduplicate files from the file system being traversed. If several filesystems are inventoried in a single True File registry, duplicatesacross the volumes are also eliminated.

(A) Traverse the underlying file system in the operating system. Foreach file encountered, excluding directories, perform the following:

-   -   (i) Assimilate the file encountered (using the Assimilate File        primitive mechanism). This process computes its True Name and        moves its data into the True File registry 126.    -   (ii) Create a pathname consisting of the path to the volume        directory and the relative path of the file on the media. Link        this path to the computed True Name using the Link Path to True        Name primitive mechanism.        2. Inventory Removable, Read-Only Files

A system with access to removable, read-only media volumes (such as WORMdisks and CD-ROMs) can create a usable inventory of the files on thesedisks without having to make online copies. These objects can then beused for archival purposes, directory overlays, or other needs. Anoperator must request that an inventory be created for such a volume.

This mechanism allows for maintaining inventories of the contents offiles and data items on removable media, such as diskettes and CD-ROMs,independent of other properties of the files such as name, location, anddate of creation.

The mechanism creates an online inventory of the files on one or moreremovable volumes, such as a floppy disk or CD-ROM, when the data on thevolume is represented as a directory. The inventory service uses a TrueName to identify each file, providing a way to locate the dataindependent of its name, date of creation, or location.

The inventory can be used for archival of data (making it possible toavoid archiving data. When that data is already on a separate volume),for grooming (making it possible to delete infrequently accessed filesif they can be retrieved from removable volumes), for version control(making it possible to generate a new version of a CD-ROM without havingto copy the old version), and for other purposes.

The inventory is made by creating a volume directory in the mediainventory in which each file named identifies the data item on thevolume being inventoried. Data items are not copied from the removablevolume during the inventory process.

An operator must request that an inventory be created for a specificvolume. Once created, the volume directory can be frozen or copied likeany other directory. Data items from either the physical volume or thevolume directory can be accessed using the Open File operating systemmechanism which will cause them to be read from the physical volumeusing the Realize True File from Location primitive mechanism.

To create an inventory the following steps are taken:

(A) A volume directory in the media inventory is created to correspondto the volume being inventoried. Its contextual name identifies thespecific volume.

(B) A source table entry 144 for the volume is created in the sourcetable 130. This entry 144 identifies the physical source volume and thevolume directory created in step (A).

(C) The filesystem on the volume is traversed. For each fileencountered, excluding directories, the following steps are taken:

-   -   (i) The True Name of the file is computed. An entry is created        in the True Name registry 124, including the True Name of the        file using the primitive mechanism. The source field of the True        Name registry entry 140 identifies the source table entry 144.    -   (ii) A pathname is created consisting of the path to the volume        directory and the relative path of the file on the media. This        path is linked to the computed True Name using Link Path to True        Name primitive mechanism.

(D) After all files have been inventoried, the volume directory isfrozen. The volume directory serves as a table of contents for thevolume. It can be copied using the Copy File or Directory primitivemechanism to create an “overlay” directory which can then be modified,making it possible to edit a virtual copy of a read-only medium.

3. Synchronize Directories

Given two versions of a directory derived from the same starting point,this mechanism creates a new, synchronized version which includes thechanges from each. Where a file is changed in both versions, thismechanism provides a user exit for handling the discrepancy. By usingTrue Names, comparisons are instantaneous, and no copies of files arenecessary.

This mechanism lets a local processor synchronize a directory to accountfor changes made at a remote processor. Its purpose is to bring a localcopy of a directory up to date after a period of no communicationbetween the local and remote processor. Such a period might occur if thelocal processor were a mobile processor detached from its server, or iftwo distant processors were run independently and updated nightly.

An advantage of the described synchronization process is that it doesnot depend on synchronizing the clocks of the local and remoteprocessors. However, it does require that the local processor track itsposition in the remote processor's audit file.

This mechanism does not resolve changes made simultaneously to the samefile at several sites. If that occurs, an external resolution mechanismsuch as, for example, operator intervention, is required.

The mechanism takes as input a start time, a local directory pathname, aremote processor name, and a remote directory pathname name, and itoperates by the following steps:

(A) Request a copy of the audit file 132 from the remote processor usingthe Request True File remote mechanism.

(B) For each entry 146 in the audit file 132 after the start time, ifthe entry indicates a change to a file in the remote directory, performthe following steps:

-   -   (i) Compute the pathname of the corresponding file in the local        directory. Determine the True Name of the corresponding file.    -   (ii) If the True Name of the local file is the same as the old        True Name in the audit file, or if there is no local file and        the audit entry indicates a new file is being created, link the        new True Name in the audit file to the local pathname using the        Link Path to True Name primitive mechanism.    -   (iii) Otherwise, note that there is a problem with the        synchronization by sending a message to the operator or to a        problem resolution program, indicating the local pathname,        remote pathname, remote processor, and time of change.

(C) After synchronization is complete, record the time of the finalchange. This time is to be used as the new start time the next time thisdirectory is synchronized with the same remote processor.

4. Publish Region

The publish region mechanism allows a processor to offer the files in aregion to any client processors for a limited period of time.

The purpose of the service is to eliminate any need for clientprocessors to make reservations with the publishing processor. This inturn makes it possible for the publishing processor to service a muchlarger number of clients.

When a region is published, an expiration date is defined for all filesin the region, and is propagated into the publishing system's True Fileregistry entry record 140 for each file.

When a remote file is copied, for instance using the Copy File operatingsystem mechanism, the expiration date is copied into the source field ofthe client's True File registry entry record 140. When the source is apublishing system, no dependency need be created.

The client processor must occasionally and in background, check forexpired links, to make sure it still has access to these files. This isdescribed in the background mechanism Check for Expired Links.

5. Retire Directory

This mechanism makes it possible to eliminate safely the True Files in adirectory, or at least dependencies on them, after ensuring that anyclient processors depending on those files remove their dependencies.The files in the directory are not actually deleted by this process. Thedirectory can be deleted with the Delete File operating systemmechanism.

The mechanism takes the pathname of a given directory, and optionally,the identification of a preferred alternate source processor for clientsto use. The mechanism performs the following steps:

(A) Traverse the directory. For each file in the directory, perform thefollowing steps:

-   -   (i) Get the True Name of the file from its path and find the        True File registry entry 140 associated with the True Name.    -   (ii) Determine an alternate source for the True File. If the        source IDs field of the TFR entry includes the preferred        alternate source, that is the alternate source. If it does not,        but includes some other source, that is the alternate source. If        it contains no alternate sources, there is no alternate source.    -   (iii) For each dependent processor in the True File registry        entry 140, ask that processor to retire the True File,        specifying an alternate source if one was determined, using the        remote mechanism.        6. Realize Directory at Location

This mechanism allows the user or operating system to force copies offiles from some source location to the True File registry 126 at a givenlocation. The purpose of the mechanism is to ensure that files areaccessible in the event the source location becomes inaccessible. Thiscan happen for instance if the source or given location are on mobilecomputers, or are on removable media, or if the network connection tothe source is expected to become unavailable, or if the source is beingretired.

This mechanism is provided in the following steps for each file in thegiven directory, with the exception of subdirectories:

(A) Get the local directory extensions table entry record 138 given thepathname of the file. Get the True Name of the local directoryextensions table entry record 138. This service assimilates the file ifit has not already been assimilated.

(B) Realize the corresponding True File at the given location. Thisservice causes it to be copied to the given location from a remotesystem or removable media.

7. Verify True File

This mechanism is used to verify that the data item in a True Fileregistry 126 is indeed the correct data item given its True Name. Itspurpose is to guard against device errors, malicious changes, or otherproblems.

If an error is found, the system has the ability to “heal” itself byfinding another source for the True File with the given name. It mayalso be desirable to verify that the error has not propagated to othersystems, and to log the problem or indicate it to the computer operator.These details are not described here.

To verify a data item that is not in a True File registry 126, use theCalculate True Name primitive mechanism described above.

The basic mechanism begins with a True Name, and operates in thefollowing steps:

(A) Find the True File registry entry record 140 corresponding to thegiven True Name.

(B) If there is a True File ID for the True File registry entry record140 then use it. Otherwise, indicate that no file exists to verify.

(C) Calculate the True Name of the data item given the file ID of thedata item.

(D) Confirm that the calculated True Name is equal to the given TrueName.

(E) If the True Names are not equal, there is an error in the True Fileregistry 126. Remove the True File ID from the True File registry entryrecord 140 and place it somewhere else. Indicate that the True Fileregistry entry record 140 contained an error.

8. Track for Accounting Purposes

This mechanism provides a way to know reliably which files have beenstored on a system or transmitted from one system to another. Themechanism can be used as a basis for a value-based accounting system inwhich charges are based on the identity of the data stored ortransmitted, rather than simply on the number of bits.

This mechanism allows the system to track possession of specific dataitems according to content by owner, independent of the name, date, orother properties of the data item, and tracks the uses of specific dataitems and files by content for accounting purposes. True names make itpossible to identify each file briefly yet uniquely for this purpose.

Tracking the identities of files requires maintaining an accounting log134 and processing it for accounting or billing purposes. The mechanismoperates in the following steps:

(A) Note every time a file is created or deleted, for instance bymonitoring audit entries in the Process Audit File Entry primitivemechanism. When such an event is encountered, create an entry 148 in theaccounting log 134 that Shows the responsible party and the identity ofthe file created or deleted.

(B) Every time a file is transmitted, for instance when a file is copiedwith a Request True File remote mechanism or an Acquire True File remotemechanism, create an entry in the accounting log 134 that shows theresponsible party, the identity of the file, and the source anddestination processors.

(C) Occasionally run an accounting program to process the accounting log134, distributing the events to the account records of each responsibleparty. The account records can eventually be summarized for billingpurposes.

9. Track for Licensing Purposes

This mechanism ensures that licensed files are not used by unauthorizedparties. The True Name provides a safe way to identify licensedmaterial. This service allows proof of possession of specific filesaccording to their contents without disclosing their contents.

Enforcing use of valid licenses can be active (for example, by refusingto provide access to a file without authorization) or passive (forexample, by creating a report of users who do not have properauthorization).

One possible way to perform license validation is to perform occasionalaudits of employee systems. The service described herein relies on TrueNames to support such an audit, as in the following steps:

(A) For each licensed product, record in the license table 136 the TrueName of key files in the product (that is, files which are required inorder to use the product, and which do not occur in other products)Typically, for a software product, this would include the mainexecutable image and perhaps other major files such as clip-art,scripts, or online help. Also record the identity of each system whichis authorized to have a copy of the file.

(B) occasionally, compare the contents of each user processor againstthe license table 136. For each True Name in the license table do thefollowing:

-   -   (i) Unless the user processor is authorized to have a copy of        the file, confirm that the user processor does not have a copy        of the file using the Locate True File mechanism.    -   (ii) If the user processor is found to have a file that it is        not authorized to have, record the user processor and True Name        in a license violation        The System in Operation

Given the mechanisms described above, the operation of a typical DPsystem employing these mechanisms is now described in order todemonstrate how the present invention meets its requirements andcapabilities.

In operation, data items (for example, files, database records,messages, data segments, data blocks, directories, instances of objectclasses, and the like) in a DP system employing the present inventionare identified by substantially unique identifiers (True Names), theidentifiers depending on all of the data in the data items and only onthe data in the data items. The primitive mechanisms Calculate True Nameand Assimilate Data Item support this property. For any given data item,using the Calculate True Name primitive mechanism, a substantiallyunique identifier or True Name for that data item can be determined.

Further, in operation of a DP system incorporating the presentinvention, multiple copies of data items are avoided (unless they arerequired for some reason such as backups or mirror copies in afault-tolerant system). Multiple copies of data items are avoided evenwhen multiple names refer to the same data item. The primitivemechanisms Assimilate Data Items and True File support this property.Using the Assimilate Data Item primitive mechanism, if a data itemalready exists in the system, as indicated by an entry in the True Fileregistry 126, this existence will be discovered by this mechanism, andthe duplicate data item (the new data item) will be eliminated (or notadded). Thus, for example, if a data file is being copied onto a systemfrom a floppy disk, if, based on the True Name of the data file, it isdetermined that the data file already exists in the system (by the sameor some other name), then the duplicate copy will not be installed. Ifthe data item was being installed on the system by some name other thanits current name, then, using the Link Path to True Name primitivemechanism, the other (or new) name can be linked to the already existingdata item.

In general, the mechanisms of the present invention operate in such away as to avoid recreating an actual data item at a location when a copyof that data item is already present at that location. In the case of acopy from a floppy disk, the data item (file) may have to be copied(into a scratch file) before it can be determined that it is aduplicate. This is because only one processor is involved. On the otherhand, in a multiprocessor environment or DP system, each processor has arecord of the True Names of the data items on that processor. When adata item is to be copied to another location (another processor) in theDP system, all that is necessary is to examine the True Name of the dataitem prior to the copying. If a data item with the same True Namealready exists at the destination location (processor), then there is noneed to copy the data item. Note that if a data item which alreadyexists locally at a destination location is still copied to thedestination location (for example, because the remote system did nothave a True Name for the data item or because it arrives as a stream ofun-named data), the Assimilate Data Item primitive mechanism willprevent multiple copies of the data item from being created.

Since the True Name of a large data item (a compound data item) isderived from and based on the True Names of components of the data item,copying of an entire data item can be avoided. Since some (or all) ofthe components of a large data item may already be present at adestination location, only those components which are not present thereneed be copied. This property derives from the manner in which TrueNames are determined.

When a file is copied by the Copy File or Directory operating systemmechanism, only the True Name of the file is actually replicated.

When a file is opened (using the open File operating system mechanism),it uses the Make True File Local primitive mechanism (either directly orindirectly through the Create Scratch File primitive mechanism) tocreate a local copy of the file. The Open File operating systemmechanism uses the Make True File Local primitive mechanism, which usesthe Realize True File from Location primitive mechanism, which, in turnuses the Request True File remote mechanism.

The Request True File remote mechanism copies only a single data itemfrom one processor to another. If the data item is a compound file, itscomponent segments are not copied, only the indirect block is copied.The segments are copied only when they are read (or otherwise needed).

The Read File operating system mechanism actually reads data. The ReadFile mechanism is aware of compound files and indirect blocks, and ituses the Realize True File from Location primitive mechanism to makesure that component segments are locally available, and then uses theoperating system file mechanisms to read data from the local file.

Thus, when a compound file is copied from a remote system, only its TrueName is copied. When it is opened, only its indirect block is copied.When the corresponding file is read, the required component segments arerealized and therefore copied.

In operation data items can be accessed by reference to their identities(True Names) independent of their present location. The actual data itemor True File corresponding to a given data identifier or True Name mayreside anywhere in the system (that is, locally, remotely, offline,etc). If a required True File is present locally, then the data in thefile can be accessed. If the data item is not present locally, there area number of ways in which it can be obtained from wherever it ispresent. Using the source IDs field of the True File registry table, thelocation(s) of copies of the True File corresponding to a given TrueName can be determined. The Realize True File from Location primitivemechanism tries to make a local copy of a True File, given its True Nameand the name of a source location (processor or media) that may containthe True File. If, on the other hand, for some reason it is not knownwhere there is a copy of the True File, or if the processors identifiedin the source IDs field do not respond with the required True File, theprocessor requiring the data item can make a general request for thedata item using the Request True File remote mechanism from allprocessors in the system that it can contact.

As a result, the system provides transparent access to any data item byreference to its data identity, and independent of its present location.

In operation, data items in the system can be verified and have theirintegrity checked. This is from the manner in which True Names aredetermined. This can be used for security purposes, for instance, tocheck for viruses and to verify that data retrieved from anotherlocation is the desired, and requested data. For example, the systemmight store the True Names of all executable applications on the systemand then periodically redetermine the True Names of each of theseapplications to ensure that they match the stored True Names. Any changein a True Name potentially signals corruption in the system and can befurther investigated. The Verify Region background mechanism and theVerify True File extended mechanisms provide direct support for thismode of operation. The Verify Region mechanism is used to ensure thatthe data items in the True File registry have not been damagedaccidentally or maliciously. The Verify True File mechanism verifiesthat a data item in a True File registry is indeed the correct data itemgiven its True Name.

Once a processor has determined where (that is, at which other processoror location) a copy of a data item is in the DP system, that processormight need that other processor or location to keep a copy of that dataitem. For example, a processor might want to delete local copies of dataitems to make space available locally while knowing that it can rely onretrieving the data from somewhere else when needed. To this end thesystem allows a processor to Reserve (and cancel the reservation of)True Files at remote locations (using the remote mechanism). In this waythe remote locations are put on notice that another location is relyingon the presence of the True File at their location.

A DP system employing the present invention can be made into afault-tolerant system by providing a certain amount of redundancy ofdata items at multiple locations in the system. Using the Acquire TrueFile and Reserve True File remote mechanisms, a particular processor canimplement its own form of fault-tolerance by copying data items to otherprocessors and then reserving them there. However, the system alsoprovides the Mirror True File background mechanism to mirror (makecopies) of the True File available elsewhere in the system. Any degreeof redundancy (limited by the number of processors or locations in thesystem) can be implemented. As a result, this invention maintains adesired degree or level of redundancy in a network of processors, toprotect against failure of any particular processor by ensuring thatmultiple copies of data items exist at different locations.

The data structures used to implement various features and mechanisms ofthis invention store a variety of useful information which can be used,in conjunction with the various mechanisms, to implement storage schemesand policies in a DP system employing the invention. For example, thesize, age and location of a data item (or of groups of data items) isprovided. This information can be used to decide how the data itemsshould be treated. For example, a processor may implement a policy ofdeleting local copies of all data items over a certain age if othercopies of those data items are present elsewhere in the system. The age(or variations on the age) can be determined using the time of lastaccess or modification in the local directory extensions table, and thepresence of other copies of the data item can be determined either fromthe Safe Flag or the source IDs, or by checking which other processorsin the system have copies of the data item and then reserving at leastone of those copies.

In operation, the system can keep track of data items regardless of howthose items are named by users (or regardless of whether the data itemseven have names). The system can also track data items that havedifferent names (in different or the same location) as well as differentdata items that have the same name. Since a data item is identified bythe data in the item, without regard for the context of the data, theproblems of inconsistent naming in a DP system are overcome.

In operation, the system can publish data items, allowing other,possibly anonymous, systems in a network to gain access to the dataitems and to rely on the availability of these data items. True Namesare globally unique identifiers which can be published simply by copyingthem. For example, a user might create a textual representation of afile on system A with True Name N (for instance as a hexadecimalstring), and post it on a computer bulletin board. Another user onsystem B could create a directory entry F for this True Name N by usingthe Link Path to True Name primitive mechanism. (Alternatively, anapplication could be developed which hides the True Name from the users,but provides the same public transfer service.)

When a program on system B attempts to open pathname F linked to TrueName N, the Locate Remote File primitive mechanism would be used, andwould use the Locate True File remote mechanism to search for True NameN on one or more remote processors, such as system A. If system B hasaccess to system A, it would be able to realize the True File (using theRealize True File from Location primitive mechanism) and use it locally.Alternatively, system B could find True Name N by accessing any publiclyavailable True Name server, if the server could eventually forward therequest to system A.

Clients of a local server can indicate that they depend on a given TrueFile (using the Reserve True File remote mechanism) so that the TrueFile is not deleted from the server registry as long as some clientrequires access to it. (The Retire True File remote mechanism is used toindicate that a client no longer needs a given True File.)

A publishing server, on the other hand, may want to provide access tomany clients, and possibly anonymous ones, without incurring theoverhead of tracking dependencies for each client. Therefore, a publicserver can provide expiration dates for True Files in its registry. Thisallows client systems to safely maintain references to a True File onthe public server. The Check For Expired Links background mechanismallows the client of a publishing server to occasionally confirm thatits dependencies on the publishing server are safe.

In a variation of this aspect of the invention, a processor that isnewly connected (or reconnected after some absence) to the system canobtain a current version of all (or of needed) data in the system byrequesting it from a server processor. Any such processor can send arequest to update or resynchronize all of its directories (starting at aroot directory), simply by using the Synchronize Directories extendedmechanism on the needed directories.

Using the accounting log or some other user provided mechanism, a usercan prove the existence of certain data items at certain times. Bypublishing (in a public place) a list of all True Names in the system ona given day (or at some given time), a user can later refer back to thatlist to show that a particular data item was present in the system atthe time that list was published. Such a mechanism is useful intracking, for example, laboratory notebooks or the like to prove datesof conception of inventions. Such a mechanism also permits proof ofpossession of a data item at a particular date and time.

The accounting log file can also track the use of specific data itemsand files by content for accounting purposes. For instance, aninformation utility company can determine the data identities of dataitems that are stored and transmitted through its computer systems, anduse these identities to provide bills to its customers based on theidentities of the data items being transmitted (as defined by thesubstantially unique identifier). The assignment of prices for storingand transmitting specific True Files would be made by the informationutility and/or its data suppliers; this information would be joinedperiodically with the information in the accounting log file to producecustomer statements.

Backing up data items in a DP system employing the present invention canbe done based on the True Names of the data items. By tracking backupsusing True Names, duplication in the backups is prevented. In operation,the system maintains a backup record of data identifiers of data itemsalready backed up, and invokes the Copy File or Directory operatingsystem mechanism to copy only those data items whose data identifiersare not recorded in the backup record. Once a data item has been backedup, it can be restored by retrieving it from its backup location, basedon the identifier of the data item. Using the backup record produced bythe backup to identify the data item, the data item can be obtainedusing, for example, the Make True File Local primitive mechanism.

In operation, the system can be used to cache data items from a server,so that only the most recently accessed data items need be retained. Tooperate in this way, a cache client is configured to have a localregistry (its cache) with a remote Local Directory Extensions table(from the cache server). Whenever a file is opened (or read), the LocalDirectory Extensions table is used to identify the True Name, and theMake True File Local primitive mechanism inspects the local registry.When the local registry already has a copy, the file is already cached.Otherwise, the Locate True File remote mechanism is used to get a copyof the file. This mechanism consults the cache server and uses theRequest True File remote mechanism to make a local copy, effectivelyloading the cache.

The Groom Cache background mechanism flushes the cache, removing theleast-recently-used files from the cache client's True File registry.While a file is being modified on a cache client, the Lock Cache andUpdate Cache remote mechanisms prevent other clients from trying tomodify the same file.

In operation, when the system is being used to cache data items, theproblems of maintaining cache consistency are avoided.

To access a cache and to fill it from its server, a key is required toidentify the data item desired. Ordinarily, the key is a name or address(in this case, it would be the pathname of a file). If the dataassociated with such a key is changed, the client's cache becomesinconsistent; when the cache client refers to that name, it willretrieve the wrong data. In order to maintain cache consistency it isnecessary to notify every client immediately whenever a change occurs onthe server.

By using an embodiment of the present invention, the cache key uniquelyidentifies the data it represents. When the data associated with a namechanges, the key itself changes. Thus, when a cache client wishes toaccess the modified data associated with a given file name, it will usea new key (the True Name of the new file) rather than the key to the oldfile contents in its cache. The client will always request the correctdata, and the old data in its cache will be eventually aged and flushedby the Groom Cache background mechanism.

Because it is not necessary to immediately notify clients when changeson the cache server occur, the present invention makes it possible for asingle server to support a much larger number of clients than isotherwise possible.

In operation, the system automatically archives data items as they arecreated or modified. After a file is created or modified, the Close Fileoperating system mechanism creates an audit file record, which iseventually processed by the Process Audit File Entry primitivemechanism. This mechanism uses the True File primitive mechanism for anyfile which is newly created, which in turn uses the Mirror True Filebackground mechanism if the True File is in a mirrored or archivedregion. This mechanism causes one or more copies of the new file to bemade on remote processors.

In operation, the system can efficiently record and preserve anycollection of data items. The Freeze Directory primitive mechanismcreates a True File which identifies all of the files in the directoryand its subordinates. Because this True File includes the True Names ofits constituents, it represents the exact contents of the directory treeat the time it was frozen. The frozen directory can be copied with itscomponents preserved.

The Acquire True File remote mechanism (used in mirroring and archiving)preserves the directory tree structure by ensuring that all of thecomponent segments and True Files in a compound data item are actuallycopied to a remote system. Of course, no transfer is necessary for dataitems already in the registry of the remote system.

In operation, the system can efficiently make a copy of any collectionof data items, to support a version control mechanism for groups of thedata items.

The Freeze Directory primitive mechanism is used to create a collectionof data items. The constituent files and segments referred to by thefrozen directory are maintained in the registry, without any need tomake copies of the constituents each time the directory is frozen.

Whenever a pathname is traversed, the Get Files in Directory operatingsystem mechanism is used, and when it encounters a frozen directory, ituses the Expand Frozen Directory primitive mechanism.

A frozen directory can be copied from one pathname to anotherefficiently, merely by copying its True Name. The Copy File operatingsystem mechanism is used to copy a frozen directory.

Thus it is possible to efficiently create copies of different versionsof a directory, thereby creating a record of its history (hence aversion control system).

In operation, the system can maintain a local inventory of all the dataitems located on a given removable medium, such as a diskette or CD-ROM.The inventory is independent of other properties of the data items suchas their name, location, and date of creation.

The Inventory Existing Directory extended mechanism provides a way tocreate True File Registry entries for all of the files in a directory.One use of this inventory is as a way to pre-load a True File registrywith backup record information. Those files in the registry (such aspreviously installed software) which are on the volumes inventoried neednot be backed up onto other volumes.

The Inventory Removable, Read-only Files extended mechanism not onlydetermines the True Names for the files on the medium, but also recordsdirectory entries for each file in a frozen directory structure. Bycopying and modifying this directory, it is possible to create an online patch, or small modification of an existing read-only file. Forexample, it is possible to create an online representation of a modifiedCD-ROM, such that the unmodified files are actually on the CD-ROM, andonly the modified files are online.

In operation, the system tracks possession of specific data itemsaccording to content by owner, independent of the name, date, or otherproperties of the data item, and tracks the uses of specific data itemsand files by content for accounting purposes. Using the Track forAccounting Purposes extended mechanism provides a way to know reliablywhich files have been stored on a system or transmitted from one systemto another.

True Names in Relational and Object-Oriented Databases

Although the preferred embodiment of this invention has been presentedin the context of a file system, the invention of True Names would beequally valuable in a relational or object-oriented database. Arelational or object-oriented database system using True Names wouldhave similar benefits to those of the file system employing theinvention. For instance, such a database would permit efficientelimination of duplicate records, support a cache for records, simplifythe process of maintaining cache consistency, providelocation-independent access to records, maintain archives and historiesof records, and synchronize with distant or disconnected systems ordatabases.

The mechanisms described above can be easily modified to serve in such adatabase environment. The True Name registry would be used as arepository of database records. All references to records would be viathe True Name of the record. (The Local Directory Extensions table is anexample of a primary index that uses the True Name as the uniqueidentifier of the desired records.)

In such a database, the operations of inserting, updating, and deletingrecords would be implemented by first assimilating records into theregistry, and then updating a primary key index to map the key of therecord to its contents by using the True Name as a pointer to thecontents.

The mechanisms described in the preferred embodiment, or similarmechanisms, would be employed in such a system. These mechanisms couldinclude, for example, the mechanisms for calculating true names,assimilating, locating, realizing, deleting, copying, and moving TrueFiles, for mirroring True Files, for maintaining a cache of True Files,for grooming True Files, and other mechanisms based on the use ofsubstantially unique identifiers.

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiments,it is to be understood that the invention is not to be limited to thedisclosed embodiment, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1. A method of distributing a data item to a plurality of computers in anetwork of computers, the method comprising: (A) dividing the data iteminto a plurality of segments; and (B) for each particular segment ofsaid plurality of segments, (b1) causing each said particular segment tobe stored on at least one computer in said network; (C) maintaining alist of computers onto which each of said segments has been stored.
 2. Amethod as recited in claim 1 wherein each segment of the plurality ofsegments is distinct from each other segment.
 3. A method as recited inclaim 1 wherein at least some of said computers in said network make uppart of a peer-to-peer network of computers.
 4. A method as recited inclaim 1 further comprising: (D) maintaining, for each computer on saidlist of computers, information relating to a measure of availability ofsaid each computer.
 5. A method as recited in claim 1 furthercomprising: for each particular segment, determining an identifier, saididentifier being determined as a function of all of the data in saidparticular segment and only the data in said particular segment, whereany two identical segments will have identical identifiers, and whereinthe step (C) of maintaining further comprises associating each of saidsegments with its corresponding identifier.
 6. A method as recited inclaim 1 wherein at least some of said plurality of segments are storedin an encoded form by compressing said at least some segments.
 7. Amethod as recited in claim 6 wherein at least some of said segments arecompressed after being sent.
 8. A method as recited in claim 1 furthercomprising: for each particular segment, determining an identifier, saididentifier being determined as a given function of the data comprisingsaid particular segment, where any two identical segments will haveidentical identifiers, and wherein the step (C) of maintaining furthercomprises associating each of said segments with its correspondingidentifier.
 9. A method of distributing a plurality of data items to aplurality of computers in a network of computers, the method comprising:(A) for each of said plurality of data items, (a1) dividing the dataitem into a plurality of segments; and (a2) for each particular segmentof said plurality of segments, (a21) causing each said particularsegment to be stored on at least one computer in said network; (B)maintaining a list of computers onto which each of said segments foreach of said plurality of data items has been stored.
 10. A method ofobtaining a data item at a computer in a network of computers, said dataitem comprising a plurality of segments, the method comprising: (A) foreach particular segment of said plurality of segments that comprise saiddata item, said computer: (a1) requesting said particular segment fromat least one of a plurality of computers in said network of computers;and (a2) obtaining said particular segment from said at least one of aplurality of computers in said network of computers.
 11. A method asrecited in claim 10 further comprising: said computer combining eachsaid particular segment to form said data item.
 12. A method as recitedin claim 10 wherein at least some of said computers in said network makeup part of a peer-to-peer network of computers.
 13. A method as recitedin claim 10 further comprising, for each said particular segment:obtaining a list of one or more computers that should have a copy ofsaid particular segment stored thereon, and wherein said step (a1) ofrequesting comprises requesting said particular segment from at least ofsaid computers that should have a copy of said particular segment storedthereon.
 14. A method as recited in claim 11 wherein said segments arestored on computers in said network in an encoded form, and wherein saidstep (a2) of obtaining, obtains the encoded form of each said particularsegment, the method further comprising: for each particular segment ofsaid plurality of segments that comprise said data item: (a3) decodingthe particular segment.
 15. A method as recited in claim 14 wherein theencoded form comprises a compressed form, and wherein step (a3)comprises decompressing the particular segment.
 16. A method as recitedin claim 10 wherein said particular segments have been stored on saidplurality of computers prior to said step (a1) of requesting.
 17. Amethod of obtaining a data item at a computer in a network of computers,said data item comprising a plurality of segments, the methodcomprising: (A) for each particular segment of said plurality ofsegments that comprise said data item: (a0) obtaining a list ofcomputers in said network of computers that should have said particularsegment; (a1) requesting said particular segment from at least onecomputer on said list of computers; and (a2) obtaining of saidparticular segment from said at least one computer on said list; and (B)combining each said particular segment to form said data item.
 18. Amethod as recited in claim 17 wherein at least some of said computers insaid network make up part of a peer-to-peer network of computers.
 19. Amethod as recited in claim 17 wherein said step (a2) obtains an encodedcopy of said particular segment, the method further comprising the stepof: (a3) decoding the encoded copy of the particular segment.
 20. Amethod as recited in claim 19 wherein said encoded copy is encoded bycompressing said particular segment, and wherein said step (a3) ofdecoding comprises decompressing said encoded copy of the particularsegment.
 21. A method of storing a data item, the method comprising: (A)dividing the data item into a plurality of parts; and (B) causing eachparticular part of said plurality of parts of said data item to bestored on at least one computer a said network of computers; (C)maintaining a list of computers in said network of computers onto whicheach of said parts has been stored.
 22. A method as recited in claim 21wherein at least some of said computers in said network make up part ofa peer-to-peer network of computers.